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ABSTRACT 

Past research has focused on many aspects of faculty evaluation from types of question to appropriate application. An 
increasing number of academic institutions are changing to web-based systems to take advantage of certain efficiencies 
in the collection process. This research compares results from traditional in-class versus online methods. No significant 
differences in mean scores were found in the majority of cases. However, when a major paper was required in the 
course, researchers discovered that student evaluations were significantly more negative. This study proposes a model to 
show importantfactors in considering a changefrom traditional pen and paper evaluations to online evaluations and 
whether the online process will have a significant effect on means scores and response rates. 


INTRODUCTION 

Educational institutions have continuously made efforts 
to determine methods of effective evaluation of faculty, 
courses, and degree programs. Although progress has been 
made, the need for continuous improvement is needed. 
Past research has focused on many aspects of faculty eval¬ 
uation, such as examination of the appropriate methods, 
questions, and sample size. Universities often administer 
student evaluations of faculty to provide feedback to fac¬ 
ulty for improvement of teaching effectiveness. Another 
stream of research assesses the validity and reliability of 
the instrument, while others focus on the appropriate ap¬ 
plication (i.e. professional development vs. evaluation of 
faculty). 

Recently, administrators have considered online student 
evaluations as opposed to the more traditional in-class, 
paper and pen type evaluations. An increasing number 
of academic institutions are considering a change to web- 
based systems to take advantage of the efficiencies in the 
collection of end-of-semester course evaluations. In con¬ 
sidering such a change, it is important that researchers de¬ 
termine whether the method will affect mean evaluation 
scores and response rates. Even though the literature has 
criticized reliance on student evaluations of faculty teach¬ 
ing, universities continue to use them to measure faculty 


effectiveness. However, since the results may also be used 
in determining faculty merit, promotion, and tenure, 
these instruments and methods are of vital importance to 
faculty. This research attempts to determine if differences 
exist in results between traditional methods and online 
evaluations and to assess the resulting implications. Also, 
this study proposes a model which shows variables, such as 
class environment which may affect the validity and reli¬ 
ability of online faculty evaluations. 

LITERATURE REVIEW 

The issue of student evaluation of teaching remains con¬ 
troversial with mixed findings. The advantages of online 
over traditional in-class, paper methods include reduced 
costs, ease of reaching representative samples, the ability 
to validate data during collection, and rapid dissemina¬ 
tion of results (Couper, 2000). Also, accounting admin¬ 
istrators found a statistically inverse relationship between 
the weight given the evaluations and the emphasis placed 
on teaching (Read, Rama, & Raghunandan, 2001). How¬ 
ever, some administrators are concerned that the misuse 
of evaluations may result in faculty members engaging in 
various activities designed specifically to affect student 
ratings on teaching evaluations, rather than to improve in¬ 
struction (Simpson & Siguaw, 2000). In addition, faculty 
may be concerned that the evaluations do not accurately 
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reflect their teaching skills. On the other hand, Hobson 
and Talbot (2001) concluded that well-developed student 
evaluations with adequate data may provide some of the 
best measures of teaching effectiveness. Nevertheless, the 
question remains as to the best method of obtaining stu¬ 
dent evaluations to obtain valid and reliable data. 

The classic method of obtaining student evaluations is the 
traditional paper and pen survey, as compared to online 
surveys. In considering a change to online course evalua¬ 
tions, researches assessed how such a change would affect 
the quality of course data. The conclusion showed that on¬ 
line evaluation methods lead to lower response rates, but 
that lower response rates do not affect mean scores. Con¬ 
sequently, a change from traditional methods to online 
evaluations is unlikely to adversely affect faculty evalua¬ 
tion scores (Avery, Bryant, Mathios, Kang, & Bell, 2006). 
Furthermore, Donmeyer, Baum, Hanna, and Chapman 
(2004) discovered that online evaluations produce es¬ 
sentially the same scores as traditional evaluations (Dom- 
meyer et al, 2004; Layne, DeCristoforo, & McGinty, 
1999). Although they found that online response rates are 
lower, the mean scores of the traditional evaluation are es¬ 
sentially the same as the online course evaluations. 

In addition, smaller classes tend to have higher response 
rates with online surveys while response rates increase 
over time. Johnson (2002) noted that response rates in¬ 
creased yearly from 40%, 51%, 62%, and finally 71% dur¬ 
ing the last year of evaluations. Another research project 
found that students who completed the online surveys 
responded more favorably toward faculty than students 
completing the paper format. Consequently, class evalu¬ 
ations results became more positive for faculty (Carini, 
Hayek, Kuh, Kennedy, & Ouimet, 2003)). Interestingly, 
when students used online surveys, they typed seven 
times more in comments than with in-class paper evalu¬ 
ations. Likewise, research in a graduate management class 
found that students typed an average of four times as 
many words online as they did using a traditional method 
(Hmieleski & Champagne, 2000). 

Anderson, Cain, and Bird (2005) found that online eval¬ 
uations are less subject to faculty influence, since students 
are allowed to have as much time as they desire with flex¬ 
ible time periods to complete the instrument. Another 
line of research examined student attitudes toward the 
methods of evaluation and discovered that the online 
method has a lower response rate than does the in-class 
method. Further, online respondents (1) expressed con¬ 
cerns about anonymity of their responses; (2) complained 
that the process was time consuming; and (3) disliked 
the complicated login procedure (Dommeyer, Baum, & 
Hanna, 2002). However, Layne et al. (1999) reported that 
students prefer completing electronic evaluations com¬ 


pared to traditional ones. Ravelli’s (2002) research project 
involving focus groups indicated that online evaluations 
are easy to use, allow for more confidentiality, and provide 
more time for comments. 

Online evaluations, such as RateMyProfessors.com, may 
provide some insight into factors affecting evaluation pro¬ 
cesses. Lawson and Stephenson (2005) discovered several 
factors that affect student perceptions of professors using 
the RateMyProfessors.com information. However, they 
concluded that students are subjective in their evaluations 
(Lawson & Stephenson, 2005). On the other hand. Kin¬ 
dred and Mohammed (2005) concluded that the student 
evaluations of professors matched their actual concerns 
regarding the quality of instruction in terms of compe¬ 
tence, knowledge, clarity, and helpfulness. Interestingly, 
they stated that personality and appearance were not as 
important. 

Other factors influencing student evaluations include 
class size, difficulty of the class, percent of students re¬ 
sponding, and length of the class. Furthermore, expecta¬ 
tions of higher grades resulted in more favorable student 
evaluations; however, the relationship is significantly dif¬ 
ferent depending upon faculty rank (Read et ah, 2001). 

METHODOLOGY 

The basic research question is as follows: Are university 
online evaluations an effective method to measure class 
and faculty effectiveness? To answer this question, classes 
were chosen in a college of business administration and 
students were asked to evaluate the class in the traditional 
method as well as online. Eight classes in a college of busi¬ 
ness administration were selected to participate in the 
study. The classes were as follows: 


MANA 3311, 
MANA 5320 

Organizational Behavior 

FINA 3311 

Principles of Finance 

MARK 3311 

Principles of Marketing 

MANA 3325 

Entrepreneurship 

MANA 4310 

International Management 

FINA 3330 

Security Analysis & Portfolio Mgmt. 


Students were asked to use the traditional evaluation 
instrument which consisted of eight questions and was 
identical to the online evaluation. The traditional pen and 
paper evaluation was performed at the end of the semes¬ 
ter as the instructor left the room and followed standard 
procedures. The administration requested students to go 
online to evaluate all of their classes, including those out- 
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side the college. Students were made aware that the online 
evaluations would be placed on the university website for 
public viewing. 

To encourage student participation and a high response 
rate, the university administration used an incentive 
where students would be able to see their final grades early 
if they completed the online class evaluations. Surprising¬ 
ly, the university received a 65% response rate in its first 
attempt to use the online system. 

Students were asked to rank eight questions on a Likert 
scale using strongly agree, agree, disagree, and strongly 
disagree (see Table 1). In assessing differences, mean scores 
for both the traditional method (pen and pencil) and on¬ 
line were calculated. T-tests were conducted to determine 
if significant differences existed between two sample 
means for each question in the eight classes. 

The following hypothesis was formulated: 

HI: No significant differences exist in traditional 
(pen and paper) method and online class eval¬ 
uations. 


FINDINGS 

Means and standard deviations for both the traditional 
and online version were obtained for each question on the 
survey (see Table 1). Next, the probability points based on 
a two-tailed Student’s ^-distribution were obtained using 
built-in Excel statistical functions. 

The results showed that no significant differences in mean 
scores existed in six of the classes. However, Table 2 shows 
that two of the classes had significantly lower online 
course evaluations (7-test values ranged from 2.23 to 7.23). 
These two classes had an end-of-semester project that had 
not yet been graded when they used the traditional evalu¬ 
ations but had been completed and graded when they 
completed the online evaluations. Figure 1 further illus¬ 
trates this phenomena, showing that factors exist that can 
seriously impact the results of student evaluation results. 

The t-test results for six of the classes indicated that no 
statistically significant difference existed between the 
mean scores for each question in the traditional evalua¬ 
tion and the online evaluation, with probabilities ranging 
from 0.07 to 1. It is also noteworthy that for the six classes 
without an end-of-semester project, the instructor and or- 
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Student Evaluations Mean Scores according to Traditional and Online Classes 
Where No Significant Differences Exist 
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Table 2 

Student Evaluations Mean Scores according to 
Traditional and Online Classes 
Where Significant Differences Exist 
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ganization of the course had the highest probability of sta¬ 
tistically significant differences, both with probabilities of 
less than 0.4. 

The remaining two classes involved in the study, however, 
showed very different results: the maximum probability 
was calculated to be 0.0043, indicating a statistically sig¬ 
nificant difference between mean values for the tradition¬ 
al and online evaluation. This phenomenon is further il¬ 
lustrated in Figure 1, which shows average probability (p) 
based on question type for classes with end-of-semester 
project and without end-of-semester project. In short, the 


results of this study indicate that, barring additional fac¬ 
tors, the results of online evaluations do not significantly 
differ from those obtained using traditional methods. 
Figure 2 shows the mean responses according to average 
probability (j >) values. 

PROPOSED MODEL AND IMPLICATIONS 

The following is a discussion of several variables, which 
may affect the success of online evaluations. Next, a pro¬ 
posed model is offered. 


Figure 1 
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Figure 2 

Mean Response for Each Question 


Average probability (p) 

3.60 3.70 3.80 3.90 4.00 4.10 4.20 4.30 4.40 4.50 4.60 


The course was well organized. 
The instructor communicated effectively. 
The instructor showed interest in progress of students, 
lests/assignments were graded and returned promptly. 
Free to ask questions, disagree & express ideas. 

Course has been of value 
Overall, this instructor was 
Overall, this course was 


i In-Class Online 


Institutional Variables 


In order to increase the response rate, incentives are used, 
such as a drawing for an iPad or priority viewing of ex¬ 
ams. If the questions are short and the process is easy to 
answer, students should be more prone to complete the 
online evaluations. The survey data show no significant 
differences between the two methods with the exception 
of those classes where a large end-of-semester project was 
involved. However, other variables which could affect the 
online effectiveness include incentives, ease of comple¬ 


tion, types of question, and grade perception as shown in 
the proposed model (Figure 3). 

A major issue in assessing the reliability of the method is 
in terms of obtaining a large participation percent. If the 
sample size is not large enough, non-response bias could 
exist. In other words, care should be taken to increase the 
sample size and include a cross section of students within 
the class. However, it is possible that students who com¬ 
plete the online evaluations are biased in that they have 


Figure 3 

Environment and Variables Affecting Online Faculty Evaluation 
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either an extremely positive or negative perception of the 
course, and the results are, therefore, skewed. 

Student/Faculty Factors 

A major finding revealed that no significant differences 
existed in the two methods with the exception of courses 
with a major project. Students may have an idea of what 
their project grade was before the evaluation. After the 
evaluation, the grade may indeed be lower and thus stu¬ 
dents gave much lower evaluations. It seems that a profes¬ 
sor must give students short-term feedback with potential 
grades during the semester in order that a student may 
have adequate assessment of his/her performance. 

Certainly, whether the course is graduate/undergradu- 
ate, online, or distance learning would have a bearing 
on the student’s evaluation. Also, students’ perception of 
anonymity and likeability may affect the results. Further¬ 
more, the size, length and difficulty of class may be vari¬ 
ables in the final evaluation. 

Benefits 

Faculty evaluations are important for a number of reasons 
including merit raises, faculty recruitment, and promo¬ 
tion and tenure decisions. For this reason, administrators 
should be cognizant of their strengths and limitations. 
Many faculty and administrators believe the major pur¬ 
pose of such evaluations is for faculty development only. 

DISCUSSION 

While faculty evaluation methods continue to expand, 
providing up-to-date, accurate information in a timely 
manner presents an additional challenge. Administra¬ 
tors should enhance their efforts to develop and coordi¬ 
nate the most efficient methods for faculty evaluation to 
achieve institutional, departmental, and faculty objec¬ 
tives. Therefore, it is vital to have accurate information to 
measure teaching effectiveness. This research shows that 
online evaluations have positive implications with certain 
variables being examined. Although no significant differ¬ 
ences existed in the majority of the classes sampled, a 3.6 
(in -class) or 3.3 (online) mean value may be different in 
the minds of the students and other viewers. Also, it is im¬ 
portant to note the type of class may be a major influence 
on evaluations. If the professor gives no feedback until 
the end of the semester, it could be a potential problem in 
overall evaluations. 

Administrators may need to enhance new methods to in¬ 
crease sample size. Various incentives should be examined, 
such as immediate access to grades or a drawing for several 
iPads. Also, faculty and staff could provide an announce¬ 


ment on Blackboard or other sites explaining the proce¬ 
dure. This communication should emphasize and clarify 
the purpose, outline, and purpose of these evaluations. 

LIMITATIONS OF RESEARCH 

The major limitation of this study was the use of only 
eight classes within one college and for one semester. Fu¬ 
ture research would do well to have a formal study of a 
large number of classes within the entire university to de¬ 
termine where differences exist. 

Information derived from this research may be used to as¬ 
sess the purpose and methodology of faculty evaluation 
and provide better insight into improving the instrument 
and overall procedure. Both methods have been exam¬ 
ined as to their role in effective teaching, but this study 
compares the results of two methods of obtaining student 
evaluations and determines if significant differences exist 
and their implications. Although other research examines 
the two methods, this research is different as faculty and 
students were aware that the results will be placed on the 
Internet for viewing. Therefore, obtaining accurate results 
appears to be even more important. 

REFERENCES 

Anderson, H. M., Cain, J., & Bird, E. (2005). Online 
course evaluations: Review of literature and a pilot 
study. American Journal of Pharmaceutical Education, 
69(1), 34-43. 

Avery, R. J., Bryant, W. K., Mathios, A, Kang, H., & Bell, 
D. (2006). Electronic course evaluations: Does an 
online delivery system influence student evaluations? 
Journal of Economic Education, 37(1) Winter, 21-37. 

Carini, R. M„ Hayek, J. C., Kuh, G. D„ Kennedy, J. M„ 
& Ouimet, J. A. (2003). College student responses to 
web and paper surveys: Does mode matter? Research in 
Higher Education, 44, 1-19. 

Couper, M. (2000). Web surveys: A review of issues and 
approaches. Public Opinion Quarterly, 64, 464-494. 

Donmeyer, C. J., Baum, R, & Hanna, R. W. (2002). Col¬ 
lege students’ attitudes toward methods of collecting 
teaching evaluations: In-class versus onlin e. Journal of 
Education for Business, 78( 1), 11-15. 

Dommeyer, C. J., Baum, R, Hanna, R. W., & Chapman, 
K. S. (2004). Gathering faculty teaching evaluations by 
in-class and online surveys: Their effects on response 
rates and evaluations. Assessment & Evaluation in 
Higher Education, 29(5), 611-623. 


16 


Fall 2013 (Volume 9 Issue 2) 



Student Evaluations of Faculty Revisited: Online Versus Traditional Methods 


Hmieleski, K., & Champagne, M. V. (2000). Plugging in 
to course evaluation. The Technology Sources Archive. 
Retrieved from http://technologysource.org/article/ 
plugging_in_to_course_evaluation/ 

Hobson, S. M„ & Talbot, D. N. (2001). Understanding 
student evaluations: What all faculty should know. 
College Teaching, 49(1), 26-31. 

Kasiar, J. B., Schroeder, S. L„ & Holstad, S. G. (2002). 
Comparison of traditional and web-based course eval¬ 
uation processes in a required, team-taught pharmaco¬ 
therapy course. American Journal of Pharmaceutical 
Education, 66, 268-270. 

Kindred, J., & Mohammed, S. N. (2005). “He will crush 
you like an academic ninja!”: Exploring teacher ratings 
on RateMyProfessors.com.y 0 z/nW of Computer-Medi¬ 
ated Communication, 10(f). 

Kuhtman, M. (2004). Review of online student ratings 
of instruction. College and University Journal, 80(1), 
64- 67. 


Lawson, R. A., & Stephenson, E. F. (2005). Easiness, at¬ 
tractiveness, and faculty evaluations: Evidence from 
RateMyProfessors.com. Atlantic Economic Journal, 
33(4), 485-486. 

Layne, B., DeCristoforo, J., & McGinty, D. (1999). Elec¬ 
tronic versus traditional student ratings of instruction. 
Research in Eligher Education, 40(2), 221-232. 

Read, W. J., Rama, D. V., & Raghunandan, K. (2001). The 
relationship between student evaluations of teaching 
and faculty evaluations. Journal of Education for Busi¬ 
ness, 76(4), 189-192. 

Simpson, P. M., & Siguaw, J. A. (2000). An exploratory 
study of the faculty response. Journal of Marketing 
Education, 22(3), 199-213. 


Journal of Academic Administration in Higher Education 


17 



